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a.  Scientific  and  Technical  Objectives 

Warfighters  who  suffer  combat,  training  or  accidental  injuries  that 
damage  their  sensory  capabilities  or  mobility  have  great  difficulty  returning 
to  productive  lifestyles  once  healed  from  the  initial  trauma.  This  project 
advanced  technologies  for  non-invasive  sensory  and  mobility  augmentation 
in  order  to  allow  these  individuals  to  regain  hope  and  social,  productive 
lifestyles.  We  addressed  the  needs  of  these  wounded  warriors  through 
evaluation  of  current  sensory  substitution  technology,  identification  of 
specific  injured  servicemember  requirements  for  sensory  augmentation, 
development  and  demonstration  of  new  sensory  substitution  techniques  and 
application  of  these  techniques  to  mobility  enhancing  devices  to  improve 
their  usability.  Specifically  we  accomplished  the  following  objectives: 

1)  Identified  sensory  substitution  requirements  specific  to  wounded 

servicemembers  for  partial  mitigation  of  vision  sensory  and  mobility 
losses. 

2)  Improved  the  usefulness  of  available  sensory  substitution  technologies 

for  injured  military  servicemembers  through  software,  hardware 
and  data  fusion. 

3)  Developed  sensory  substitution  systems  for  warfighters  suffering  from 

loss  of  vision,  balance  and  hearing. 

4)  Developed  sensory  substitution  enabled  assistive  mobility  devices  to 

provide  functional  restoration  for  servicemembers  with  mobility 
loss. 

5)  Investigated  novel  sensory  substitution  interface  technologies. 
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b.  Approach 

Advances  in  warfighter  personal  protection  and  combat  casualty  care 
have  markedly  improved  battlefield  survivability  rates  for  those  wounded  in 
action.  The  traumatic  brain  injury  and  somatic  polytrauma  suffered  by  this 
growing  population  may  evolve  over  many  months  post  trauma  and  manifest 
significant  loss  of  one  or  more  sensory  channel.  Specific  areas  of  the  brain 
(e.g.,  the  visual  cortex)  receive  information  from  specific  sensory  organs 
(e.g.,  the  eyes)  in  the  form  of  pulses  carried  by  afferent  nerves.  Sensory 
feedback  from  these  organs,  associated  with  interaction  with  the 
environment,  guides  and  reinforces  sensory  perception.  Because  perception 
takes  place  in  the  brain,  not  at  the  end  organ  (Bach-y-Rita,  1972),  the  brain 
can  reinterpret  signals  from  specific  nerves  with  appropriate  sensory 
feedback.  While  non-invasive  sensory  substitution  technologies  do  not 
provide  replacement  for  lost  sensory  capabilities,  they  can  augment  residual 
sensation.  This  iterative,  interdisciplinary  research  and  development  project 
dramatically  improved  sensory  substitution  interface  and  mobility  assist 
exoskeleton  technologies.  Through  sensory  substitution  device 
demonstrations,  we  solicited  subjective  situation  awareness  feedback  from 
injured  servicemembers  and  identified  technology  requirements  specific  to 
the  needs  of  this  population. 

c.  Concise  Accomplishments 

During  this  project  (16  April  2007  -  31  December  2008),  we  conducted 
organizational  meetings  that  included  a  site  visit  by  the  ONR  program 
manager.  We  executed  subcontracts  to  Wicab,  Inc.,  (Middleton,  WI)  and 
ForeThought  Development,  LLC,  (Blue  Mounds,  WI)  to  initiate  development 
of  modernized  electrotactile  interfaces.  We  presented  project  overview,  goals 
and  technology  demonstrations  to  the  Commander  U.S.  Army  Medical 
Command/Acting  The  Surgeon  General,  MG  Gale  Pollock.  We  demonstrated 
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the  BrainPort®  (Wicab,  Inc.,  Middleton,  WI)  vision  system  that  interfaces  to 
the  tongue,  as  well  as  IHMC's  prototype  peripheral  vision  tactile  torso 
interface  to  three  blind  veterans  and  one  active  duty  blind  servicemember. 
An  additional  veteran  with  traumatic  brain  injury  (TBI)  and  two  civilians,  all 
with  partial  visual  impairment,  evaluated  the  vision  sensory  substitution 
systems.  The  servicemember  with  TBI  also  evaluated  the  passive  mobility 
assist  exoskeleton,  which  enabled  him  to  walk  at  a  normal  gait,  unassisted, 
for  the  first  time  since  emerging  from  his  coma.  We  prototyped  an 
electrotactile  hearing  aid  and  an  integrated  multisensory  balance 
augmentation  system.  We  provided  the  Naval  Medical  Center-San  Diego  and 
the  Atlanta  Veterans  Affairs  Hospital  with  Wicab,  Inc.,  balance  prosthetic/ 
therapy  devices  for  use  in  their  protocols  with  veteran/active  duty 
servicemembers  with  balance  disorders  (including  TBI). 

d.  Expanded  Accomplishments 

Sensory  substitution  refers  to  the  remapping  of  sensory  data  from  one 
sensory  capability  to  other  channels  of  information  perception.  Sensory  loss 
resulting  from  congenital  defect,  environmental  exposure  or  trauma  can  be 
mitigated  by  appropriate  use  of  sensory  substitution  interfaces  that  present 
data  using  intact  sensory  pathways.  The  plasticity  of  the  brain  allows  a  user 
to  learn  to  perceive  the  substituted  data  with  little  cognitive  effort.  For 
maximal  benefit,  the  interface  must  intelligently  pre-process  the  incoming 
data  to  account  for  differences  in  capability  between  the  alternative  channels 
and  the  ones  normally  used  to  perceive  given  sensory  data.  Many  devices 
and  approaches  for  sensory  substitution  have  been  developed,  and  the 
concept  itself  is  a  century  or  more  old  (Machts,  1920).  Modern  computer  and 
electronic  design,  however,  has  now  enabled  the  development  of  intelligent, 
non-invasive,  human-centered  interfaces  unobtrusive  enough  to  be  used  in 
everyday  activities.  During  this  project,  we  advanced  these  existing 
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technologies,  creating  sensory  substitution  interfaces  that  could  mitigate 
sensory  and  mobility  loss.  All  components  of  this  project  related  to 
technology  development  and  application.  The  four  areas  of  effort  included: 
sensory  substitution  interfaces  (SSI),  mobility  assist  exoskeletons  (MAE), 
process  integrated  mechanisms  (PIM)  and  electrochemical  sensory  displays. 
Sensory  substitution  prototypes  were  developed  for  components  of  vision, 
balance  and  audition.  Wearable  exoskeleton  devices,  including  powered 
exoskeletons  with  custom  designed  compliant  rotary  joint  actuators  and 
passive  exoskeletons  that  mimic  natural,  energy  efficient  human  gait  and 
weight  bearing  were  developed  for  mobility  assistance.  Preliminary 
investigations  were  performed  to  evaluate  possible  candidate  chemicals  for 
electrochemical  transmission  of  signals  and  to  determine  issues  associated 
with  network  bandwidth  usage  by  the  multiple  sensory  and  display  software 
components  in  the  sensory  substitution  and  exoskeleton  systems. 

Development 

Sensory  Substitution  Interface  (SSI) 

IHMC  completed  two  major  tasks  in  SSI  technology  development: 
device  development  and  interface  integration.  For  device  development  we 
contracted  two  companies  with  prior  expertise  in  development  of 
electrotactile  interfaces,  Wicab,  Inc.  (Middleton,  WI)  and  ForeThought 
Development,  LLC  (Blue  Mounds,  WI).  Each  of  these  companies  had 
previously  developed  prototype  and  research  devices  based  on  technologies 
originally  invented  by  the  late  Paul  Bach-y-Rita  and  his  team  at  the 
University  of  Wisconsin.  These  products  were  advanced  as  part  of  this 
project.  In  addition,  IHMC  advanced  an  in-house  designed  vibrotactile 
interface.  These  three  devices  comprised  the  tactile  display  technologies 
developed  for  use  as  SSIs.  Additionally,  IHMC  implemented  a  twelve  channel 
surround  sound  processor  system  that  provides  directional  (spatial)  audio. 
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The  SSIs  were  used  to  develop  systems  that  could  provide  sensory 
substitution  to  wounded  servicemembers  with  vision,  hearing  or  balance  loss 
or  various  combinations  thereof. 


Tactile  Interfaces 

Wicab,  Inc.,  BrainPort®  device:  The  BrainPort®  devices  available  at  the 
start  of  the  project  consisted  of  a  10x10  tongue  array.  An  earlier  prototype 
employed  a  12x12  array  that  was  deemed  too  large  for  some  users.  While 
the  100  tactile  transducer  (tactor)  array  suffices  for  balance  sensory 
substitution,  the  low  spatial  resolution  does  not  support  vision  or  hearing 
substitution  well.  As  part  of  this  project,  Wicab,  Inc.,  increased  the  resolution 
of  the  arrays  to  18x18  and  finally  to  25x25  electrodes  while  maintaining  the 
same  array  size  (Figure  1). 


Figure  1:  Wicab,  Inc.,  BrainPort®  Intra-Oral  Device  (IOD)  progression,  right  to  left,  10x10 
and  12x12  low  density  IODs  available  at  the  beginning  of  this  project,  18x18  medium 
density  IOD  used  for  demonstrations  in  this  project  and  25x25  high  resolution  IOD  delivered 
at  the  end  of  the  project. 


The  smaller,  more  densely  packed  arrays  in  these  prototypes  increased 


performance  of  the  BrainPort®  concept  sufficiently  to  prove  its  usefulness  for 
vision  and  hearing  applications.  Improvements  included:  moving  more 


circuitry  into  the  Intra-Oral  Device  (IOD)  to  minimize  cable  size,  revising  the 
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IOD  assembly  process  to  reduce  electrode  erosion  by  capping  the  gold 
electrodes  with  316LVM  stainless  steel,  encapsulating  the  IOD  in  epoxy  resin 
to  reduce  moisture  intrusion  into  the  circuitry  and  reducing  the  system 
power  consumption  to  lower  system  weight.  These  project  developments 
have  been  incorporated  into  the  newest  version  of  the  BrainPort®  vision 
device  now  available  for  clinical  research  investigations. 


Figure  2:  From  left  to  right,  Forethought  Development,  LLC,  VideoTact™  array,  worn  by 
itself  on  the  abdomen  and  worn  in  conjunction  with  the  BrainPort®  vision  system. 

ForeThought  Development,  LLC,  had  developed  their  original 
VideoTact™  abdominal  array  in  the  1990s  to  provide  a  bridge  for  the  blind  as 
personal  computer  user  interfaces  moved  from  textual  to  graphical 
paradigms.  While  a  number  of  their  arrays  were  produced,  the  decade  old 
technology  relied  on  computer  interfaces  and  integrated  circuits  which  are 
no  longer  available  or  in  common  use.  For  this  project,  ForeThought 
Development  created  a  modified  version  of  their  original  VideoTact  that 
employs  a  modern  Ethernet  (TCP/IP)  control  interface  as  well  as  packaging 
changes  to  make  the  system  more  compatible  with  IHMC's  tactile  torso 
interface  (TTI)  and  the  BrainPort®  systems  (Figure  2).  Moving  from  a 
dedicated  16  bit  PCI  card  interface  to  an  Ethernet  interface  minimizes  the 
weight  of  the  cable  that  tethers  the  unit  (and  the  user)  to  the  computer  and 
allows  any  PC  to  control  the  interface  (e.g.,  laptops,  tablets,  ultra  mobile 
PCs,  etc.).  These  smaller  PCs  can  be  worn  on  the  body,  making  the  system 
fully  wearable  for  navigation  in  free  space.  The  techniques  developed  in  this 
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project  were  used  to  set  the  requirements  for  the  next  generation  system 
that  will  be  miniaturized,  untethered  and  wearable  under  the  user's  clothes. 

The  IHMC  TTI  system  (Figure  3)  is  an  in-house  developed 
implementation  of  the  U.S.  Navy/NASA/IHMC  Tactile  Situation  Awareness 
System  (TSAS),  a  novel  tactile  situation  awareness  (SA)  display  designed  for 
aviation  (Raj,  Kass  &  Perry,  2000).  For  this  project,  it  was  updated  with  new 
printed  circuit  boards  to  allow  remote  digital  enable/disable  control  of 
individual  factors  (a  function  that  was  previously  controlled  only  by  physical 
switches).  This  allows  for  selective  disabling  of  factors  by  the  control 
software.  This  also  ensures  perception,  even  in  a  location  with  a  broken 
factor,  through  the  spatial  summation  provided  by  activating  neighboring 
factors  at  appropriate  intensities.  The  mechanism  of  donning  and  doffing  the 
TTI  garment  was  simplified  and  a  second  garment  was  assembled  to  enable 
two  individuals  to  use  the  system  simultaneously. 


Figure  3:  From  left  to  right,  IHMC's  TTI  control  hardware  and  the  TTI  garments.  Front  panel 
lights  indicate  left  TTI  channels  1-4  manually  disabled  (no  illumination)  and  right  TTI 
channels  1-4  disabled  remotely  via  software  (blue).  Enabled  tactors  (green)  turn  red  during 
thermal  shutdown  and  yellow  when  driving  a  bipolar  waveform. 

Comments  made  by  blind  demonstration  participants  indicated  some 
limitations  to  video  camera  based  sensory  substitution.  For  instance,  the 
blind  do  not  always  turn  on  room  lights  and,  while  they  can  easily  learn  to 
control  the  camera  field  of  view,  the  requirement  to  actively  adjust  zoom 
prevents  the  natural  simultaneous  perception  of  foveal  and  peripheral  vision 
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used  by  sighted  individuals.  In  particular,  touching  a  recently  blinded 
individual  without  verbal  notification  (e.g.,  to  tap  his  or  her  shoulder)  can 
lead  to  a  startle  response  as  he  or  she  can  not  perceive  anyone  approaching. 
To  provide  simultaneous  peripheral  and  foveal  vision  substitution,  IHMC 
designed  and  breadboarded  a  sensor  that  could  provide  360  degree 
peripheral  vision  via  the  TTI.  It  uses  conventional  infrared  (IR)  light  emitting 
diode  (LED)  transmitters  and  television  remote  detector  parts  spread  over 
twenty  four  emitter/receiver  pairs  evenly  distributed  and  mounted  to  a  cap 
worn  by  the  user  (Figure  4). 


Figure  4:  left  to  right,  cap  bearing  IR  emitter/receiver  modules;  LEDs  illuminate 
sequentially;  LEDs  driven  by  a  38KHz  pulsed  variable  chirp  signal;  IR  detectors  receive 
reflected  38KHz  IR  signals;  Printed  circuit  board  to  implement  24  channel  detector;  TTI 
representation  of  detected  signals. 


Initial  tests  indicate  that  the  circuit  provides  accurate  detection  of  IR 
reflective  objects  in  a  range  from  2  inches  to  15  feet.  Distance  is  correlated 
with  time  by  capturing  returns  from  a  'chirp'  illumination  signal  using  64 
time-sequential  24-bit  frames.  Linear  and  inverse  square-law  power  output 
can  be  alternated  for  near  and  far  field  detection,  respectively.  A  PC-104 
form-factor  multilayer  circuit  board  was  designed  to  provide  this  functionality 
in  a  modular,  small  package  suitable  for  testing  as  a  body  worn  device 
(Figure  4).  Software  determines  which  LED  caused  the  reflection  as  well  as 
its  intensity  at  the  time  of  detection  to  estimate  range.  This  provides 
azimuth,  elevation  and  range  to  physical  objects  in  the  environment.  Future 
development  will  include  moving  the  software-based  algorithm  into  a  Field 
Programmable  Gate  Array  (FPGA)  chip.  This  implementation  should  improve 


8 


performance  by  allowing  a  rapid  segregation  of  detected  movements  that 
result  from  self-motion  from  those  due  to  objects  moving  in  the 
environment.  This  will  pre-filter  the  data  so  that  minor  head  movements  do 
not  cause  excessive  TTI  activity  and  to  increase  the  saliency  of  objects 
moving  in  the  environment. 

Dynamic  Posture  and  Balance  System 

The  current  BrainPort®  Balance  Device  prosthetic  (Figure  5)  can 
effectively  substitute  for  lost  vestibular  (i.e.,  otolith)  organ  function  by 
providing  missing  balance  organ  cues.  Using  a  triaxial  accelerometer 
embedded  in  the  intra-oral  device,  this  system  displays  head  tilt  as  a  tactile 
stimulus  that  deviates  from  the  center  toward  the  edge  of  the  array  in  two 
degree  increments  up  to  ten  degrees  (heathy  individuals  with  a  normal 
upright  posture  will  fall  if  their  tilt  angle  exceeds  approximately  12.5 
degrees).  As  patients  progress  through  a  three  month  rehabilitation 
program,  however,  they  rapidly  learn  to  utilize  other  balance  system  cues. 
After  completing  the  training  protocol,  users  with  vestibular  deficits  no 
longer  need  the  BrainPort®  Balance  Device  and  can  return  to  many  of  their 
previous  activities  of  daily  living  (ADLs).  Users  may  gain  this  lasting  effect  by 
incorporating  data  from  the  remaining  portions  of  the  balance  system, 
namely  the  somatosensory  sense  (touch,  muscle  stretch  receptors,  joint 
position,  etc.)  and  vision,  as  well  as  any  residual  vestibular  function. 

Loss  of  the  vestibular  organ  often  occurs  rapidly  following  trauma, 
surgery  or  exposure  to  ototoxic  substances  (e.g.,  antibiotics)  and  the 
confusing  sensations  may  overwhelm  the  individual  and  instill  a  fear  of 
activity  due  to  falls  and  paroxysmal  nausea.  Standard  vestibular 
rehabilitation  physical  therapy  consists  of  progressively  more  provocative 
perturbing  maneuvers  to  stress  the  patient's  balance  and  desensitize  them 
from  nausea.  The  BrainPort®  Balance  Device  may  mitigate  the  fear  of  falling 
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during  rehabilitation,  allowing  more  rapid  sensorimotor  reorganization  for 
integration  of  the  alternate  sensory  cues. 

Unfortunately,  not  all  balance  disorders  arise  solely  from  vestibular 
deficit.  Traumatic  brain  injury,  peripheral  neuropathy,  spinal  injury  and 
amputation(s)  may  also  dramatically  affect  balance.  In  addition,  the 
BrainPort®  device  provides  veridical  balance  information  only  when  the  user 
stands  upright,  with  his  or  her  feet  inline  with  and  below  the  shoulders.  For 
example,  an  individual  leaning  over  to  pick  up  a  dropped  item  may  change 
stance  and  remain  completely  stable,  but  the  BrainPort®  would  only  indicate 
(on  the  tongue  array)  that  the  individual's  head  has  tilted  beyond  degrees. 


Figure  5:  left  to  right,  BrainPort®  Balance  Device;  Visual  display  of  insole  load  cell  activity; 
and  Dynamic  balance  assistance  prototype.  Graphic  (green  avatar)  indicates  that  user  (right 
most  inset)  has  a  stable  stance  as  the  CG  (small  magenta  dot)  remains  within  her  center  of 
pressure  stability  polygon  (projected  below  the  avatar's  feet).  CG  deviations  outside  the  box 
are  presented  to  the  user  tactually. 

In  order  to  provide  a  more  fully  functional  and  dynamic  balance 
sensory  substitution  system,  IHMC  created  a  system  (Figure  5)  that 
accurately  determines  both  the  center  of  pressure  between  the  user's  feet 
and  his  or  her  center  of  gravity  (CG).  As  long  as  the  center  of  gravity 
remains  over  the  center  (within  12.5  degrees  of  the  edges  of  the  user's 
contact  patch  with  the  ground),  he  or  she  should  be  able  to  balance  without 
changing  stance.  This  system  employs  custom  designed  insole  pressure 
transducer  arrays  with  132  load  cells  in  each  insole  (Pressure  Profile 
Systems,  Inc.,  San  Diego,  CA)  and  a  full  body,  wearable  motion  capture 
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system  (ShapeWrapII,  Measurand,  Inc.,  Fredericton,  NB).  The  insoles 
determine  center  of  pressure  under  each  foot  while  the  motion  capture 
system  determines  the  position  of  each  foot  relative  to  the  torso  to 
determine  the  convex  polygon  in  which  the  CG  must  remain  for  stability. 
Using  a  look-up  table  of  average  mass  for  hands,  forearms,  head,  torso,  legs 
and  feet,  the  system  calculates  the  moment  of  each  appendage  to  estimate 
the  three  dimensional  location  of  the  CG  (Figure  5).  The  angle  relative  to  the 
contact  patch  polygon  can  represent  deviations  from  stable  stances  tactually 
on  the  BrainPort®'  VideoTact  or  TTI.  The  ShapeWrapII  system  and  the 
Pressure  Profile  system  both  use  wireless  radio  frequency  communications 
(802.11b  and  Bluetooth,  respectively)  to  transmit  data  to  the  control 
computer  running  IHMCs  AMI  balance  agent  software,  allowing  the  user  to 
move  freely  indoors  or  outside  during  ADLs. 

Hearing  Substitution 

IHMC  developed  a  prototype,  wearable  stereo  hearing  substitution 
system  that  presents  a  stereo  tactile  audio  spectrograph  on  the  tongue.  This 
can  enhance  the  saliency  of  human  speech  in  the  ambient  acoustical 
environment  and  can  be  implemented  using  either  the  VideoTact®  or  the 
BrainPort®  (Figure  6).  The  system  decomposes  audio  from  two  microphones, 
computer  generated  audio  and  pre-recorded  audio  files  into  frequency 
spectrograms  that  are  presented  visually  and  tactually.  By  implementing 
software  voice  recognition  techniques,  the  system  identifies  components  of 
the  audio  stream  that  contain  human  speech  phonemes  and  selectively 
increases  the  amplitude  of  those  sounds.  This  increases  the  power  of  the 
phoneme  frequency  in  the  spectrogram,  enhancing  the  salience  of  human 
speech  in  the  ambient  auditory  environment. 
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Figure  6:  left,  BrainPort®  display  showing  stereo  presentation  of  a  sound  containing  a  short 
"a"  with  augmentation  of  the  "a"  frequencies'  power;  right,  Auditory  sensory  substitution 
training  system  for  VideoTact  showing  pre-recorded  words  used  for  training  and  recognition. 


During  training,  the  system  displays  a  word  as  text  and  as  both  visual 
and  tactile  spectrograms.  After  training,  the  computer  monitor  is  blanked, 
and  the  system  displays  only  the  tactile  spectrogram  of  each  word  to  the 
participant.  Initial  evaluations  with  a  limited  vocabulary  (22  words 
representing  different  phonemes)  indicate  that  users  can  reach  70%  correct 
recognition  with  20  minutes  of  training.  Future  development  will  focus  on 
improving  stereo  perception  for  locating  sound  sources  and  improving 
speech  perception.  A  tactile  hearing  aid  could  eventually  assist 
servicemembers  who  cannot  benefit  from  a  cochlear  implant.  Signal 
processing  software  could  also  be  developed  to  enhance  localization  and 
identification  of  particular  sounds  (e.g.,  phones,  car  horns,  fire  alarms,  etc.). 


Software  Integration 

The  above  systems  utilized  IHMC's  Adaptive  Multiagent  Integration 
(AMI)  architecture  to  integrate  previously  existing  software  agents  rapidly 
with  new  equipment.  AMI  uses  standardized  Java  (Sun  Microsystems,  Inc, 
Santa  Clara,  CA)  programming  techniques.  Explicit  ontological  definitions 
associated  with  each  agent  allow  rapid  integration  of  new  agents  into  the 
architecture.  This  is  possible  because  the  system  makes  data  connections 


12 


between  agents  automatically  based  on  data  types  and  relative  quality  of 
similar  data  streams.  A  large  complement  of  devices,  including  video,  audio, 
pressure,  orientation  and  psychophysiologic  sensors,  have  already  been 
integrated  as  software  agents  into  this  Adaptive  Multi-agent  Integration 
(AMI)  architecture  (Johnson,  Kulkarni,  Carff,  Raj  &  Bradshaw,  2005),  as  have 
the  original  BrainPort®  and  TSAS  interfaces.  The  architecture  is  inherently 
scalable,  using  available  processing  power  on  wired  or  wireless  networks. 
AMI  allowed  us  to  send  sensor  signals  to  multiple  displays  across  different 
modalities  (e.g.,  video,  audio,  vibrotactile,  electrotactile,  etc.) 
simultaneously  or  separately  to  evaluate  various  sensory  substitution 
implementations. 

The  AMI  architecture  supports  hierarchical  data  reduction,  conditioning 
and  polishing  such  that  at  each  level  of  reduction,  suites  of  sensor  data  can 
be  estimated.  This  significantly  improves  usability  by  providing  feedback 
without  overwhelming  the  user  with  data  (Drakunov  &  Meystel,  1998).  For 
example,  at  the  sensor  input  level,  readings  are  inherently  noisy  but  exhibit 
a  systematic  pattern  of  distortion  that  can  be  used  to  "polish"  out  the  errors 
(Teng  1999;  2004).  Observer  agents  in  AMI  (with  both  linear  and  non-linear 
explicitly  defined  system  and  subsystem  models)  provide  continuous  data 
estimates  to  replace  intermittent  sensor  data  loss  or  corruption  (DeCarlo, 
Zak  &  Drakunov,  1996;  Drakunov,  1992).  Observers  operate  as 
computational  "mental  models"  to  provide  estimates  of  missing  or  erroneous 
data  based  on  available  data.  An  example  of  how  AMI  works  follows. 

Using  the  AMI  architecture,  we  implemented  a  system  that  enabled 
blind  users  the  ability  to  interact  with  a  graphical  user  interface  (GUI)  while 
controlling  sensory  substitution  tactile  displays  with  their  intact  extraocular 
muscles  (even  with  prosthetic  eyes).  Correctly  estimating  the  user's  visual 
point  of  regard  requires  knowledge  of  eye  position  relative  to  the  head,  the 
user's  head  position  in  space  and  calibration  with  respect  to  the  environment 
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or  area  of  interest  (e.g.,  the  GUI).  The  mathematical  and  geometrical 
foundation  of  gaze  tracking  uses  Euler  angles  and  quaternions.  The  Euler 
angle  terms  correspond  to  rotations  (Euclidean  frame  of  reference)  relative 
to  the  local  base  of  a  plane.  The  quaternion  equations  extend  the  concept  of 
rotation  from  3  dimensions  to  4  dimensions,  allowing  determination  of  more 
precise  angular  position  by  avoiding  singularities  inherent  to  Euler  angle 
calculations  that  cause  "gimbal  lock"  in  aircraft. 

Our  gaze  estimation  (Figure  7)  implementation  uses  successive  frame 
transforms  based  on  quaternions  and  is  calculated  using  Java3D  libraries. 
For  this  project,  we  wrote  low  level  code  using  Java  Native  Interface  (JNI) 
methods  to  access  data  from  manufacturer  supplied  device  drivers.  Java  AMI 
software  agents  were  then  created  to  call  the  JNI  code  for  the  head  tracker 
and  eye  tracker  systems.  Simple  data  polishing  performed  in  these 
algorithms  reduce  jitter  and  filtered  out  eye  blinks.  A  third  agent  combines 
the  input  from  these  agents  to  estimate  gaze  (eye  point  of  visual  regard) 
which  provides  an  additional  level  of  smoothing. 


Figure  7:  left,  Schematic  of  visual  substitution  of  gaze  via  the  BrainPort®;  right,  Calibration 
procedure  to  register  gaze  to  screen  coordinates. 

A  nine  point  procedure  calibrated  the  agents  relative  to  a  computer 
monitor  (detents  were  created  along  the  bezel  of  the  monitor  and  blind  users 
were  instructed  to  imagine  looking  at  their  finger  tip  when  they  touched 
each  detent).  This  gaze  position  estimate  provides  the  x,  y  and  z  coordinates 
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of  the  point  of  regard  relative  to  the  screen.  This  then  allows  another  agent 
to  take  a  screen  shot  (100-200  pixels)  at  the  gaze  point  and  send  it  to  the 
Tongue  Display  Unit  (TDU)  Agent,  which  grayscales  and  normalizes  the 
image  before  transmitting  it  to  the  BrainPort®  (Figure  8). 


Figure  8:  left,  Gaze  position  marked  by  black  reticle  and  sampled  (inset)/  right,  Visual 
representation  of  tactile  image  sent  to  BrainPort®  based  on  gaze  position. 

One  drawback  to  multi-agent  systems  relates  to  their  network 
bandwidth  usage.  This  increases  with  the  number  of  agents  and  the  rate  of 
data  signals  (e.g.,  when  using  video  and  audio  streams).  The  process 
integrated  mechanism  (PIM)  concept  was  evaluated  as  a  method  of 
managing  network  usage  to  ensure  timely  delivery  of  data  to  all  displays. 
PIMs  operate  as  a  single  collaborating  system  by  virtue  of  a  single  program 
that  moves  rapidly  between  all  the  available  processors  in  the  system  (wired 
and  wirelessly  networked).  Using  a  single  program  greatly  simplifies  the 
complexity  of  developing  a  solution  for  coordination  and  collaboration  while 
at  the  same  time  improving  the  ability  to  debug,  verify,  and  validate  the 
solution  and  ensuring  predictable  network  traffic.  Only  the  program's  state 
variables  pass  over  the  network  from  node  to  node  as  each  processor 
performs  its  operations  in  turn.  This  makes  the  network  traffic  consistent 
and  manageable,  even  with  high  data  rates  and  multiple  agents.  Our 
approach  encapsulates  related  agents  as  single  PIM  nodes,  such  that  agents 
related  to  a  given  function  operate  normally  in  AMI,  but  the  top  level  (the 


15 


Gaze  Agent,  in  this  example)  would  connect  to  other  agents  (e.g.,  the 
Screen  Shot  Agent,  the  BrainPort®  Agent,  etc.)  through  a  PIM  architecture. 
This  allows  access  to  each  high  level  function  in  the  overall  architecture 
without  flooding  the  network  with  only  locally  relevant  traffic.  The  observer 
and  PIM  approaches  can  improve  robustness,  resiliency,  predictability  and 
efficiency  in  complex  sensory  substitution  systems. 

Electrochemical  interfaces 

The  current  BrainPort®  electrotactile  array  electrode  size  and 
placement  density  has  reached  the  limit  of  industrial  flexible  circuit 
manufacturability.  We  reviewed  a  number  of  alternative  concepts  for  creating 
smaller,  more  densely  packed  electrodes  to  provide  higher  resolution 
displays.  These  included  magnetic  patches  coated  with  an  enzyme  that 
diffuses  into  the  tissue  to  stimulate  the  nerves,  electrically  actuated 
microfluidic  arrays  that  release  neurotransmitters  (Glutamate,  Kainate, 
Quisqualate,  Alpha-amino-3-hydroxy-5-methyl-4  isoazolepropionic  acid,  N 
methyl  D  aspartate,  etc.)  to  stimulate  synapses  directly  and  nanowire  arrays 
for  nanoscale  electrochemical  stimuli.  Individually  addressable  nanowires 
would  provide  the  high  spatial  resolution  and  high  sensitivity  needed  to 
maximize  resolution  and  could  result  in  more  effective  chronic  neural 
interfaces.  Exploiting  these  technologies  was  outside  this  projects  scope. 

Mobility  Assist  Exoskeletons  (MAE) 

Commonly,  there  are  two  approaches  to  exoskeleton  design:  the  first 
is  pseudo  anthropomorphic;  the  second  is  anthropomorphic,  where  the  joints 
are  co-located  as  close  as  possible  and  where  there  are  connection  points  at 
the  thigh  and  shank  as  well  as  the  foot.  Although  many  exoskeletons  have 
been  developed,  most  are  designed  for  performance  (strength  and 
endurance)  augmentation  and  have  been  pseudo-anthropomorphic;  the 
exoskeleton  and  the  user's  joints  do  not  coincide  exactly  and  the  only 
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connection  point  is  at  the  end  of  the  limb,  making  them  unsuitable  for 
applications  where  the  exoskeleton  must  move  the  user's  limbs  to  ensure 
normal  gait.  IHMC  designed  and  prototyped  two  anthropomorphic  Mobility 
Assist  Exoskeleton  concepts  designed  for  those  with  weakness,  loss  of 
sensation  or  paralysis  of  one  or  both  lower  extremities.  One  prototype 
passively  enforces  normal  gait  trajectories  using  compound  joints  with 
multiple  degrees  of  freedom  that  mimic  knee  and  ankle  motion.  This  device 
also  passively  locks  the  knee  and  ankle  joints  when  weight  is  applied  by  the 
user.  IHMC  also  developed  powered  exoskeletons  that  can  track  user 
initiated  movements  and  augment  strength  (for  those  with  lower  extremity 
weakness)  or  move  the  user's  legs  in  a  normal  gait  pattern  (for  those  with 
paralysis).  Walking  in  these  exoskeletons  loads  the  user's  legs  with  his/her 
body  weight  which  should  reduce  the  rate  of  muscle  and  bone  degradation 
associated  with  loss  of  ambulation.  A  return  to  an  upright  posture  also 
enhances  cardiovascular  fitness  and  helps  with  ADLs  and  social  interactions 
which  evolved  around  bipedal  ambulation. 

Passive  Dynamic  MAE 

The  Passive  Dynamic  MAE  seeks  to  meet  the  immediate  functional  and 
ongoing  therapeutic  need  of  persons  with  difficulty  walking  due  to  paresis, 
weakness  or  atrophy  of  the  leg  muscles,  or  due  to  partial  or  total  paralysis 
from  the  waist  down.  The  passive  dynamic  mobility  exoskeleton  developed 
by  IHMC  (Figure  9)  provides  support  for  normal  gait  and  posture  by 
mimicking  the  highly  efficient  natural  motion  of  the  legs.  The  MAE 
mechanical  joints  follow  all  movements  of  the  legs  with  minimal  interference 
and  dynamically  adjust  to  the  user's  posture  and  gait.  The  joints  lock 
automatically  with  body  weight  shifts  using  artificial  "tendons"  to  support  the 
user.  The  MAE  supports  walking  without  undue  exertion  or  discomfort  to 
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prevent  musculoskeletal  deterioration  and  other  complications  associated 
with  inactivity  and  lack  of  lower  extremity  cyclic  weight  bearing. 


Figure  9:  Initial  passive  dynamic  MAE  (left  &  center);  revised  passive  dynamic  MAE  (right). 

Using  a  passive  dynamic  MAE  could  increase  ambulatory  activity  to 
support  sensorimotor  reorganization  and  cardiovascular  fitness  during 
rehabilitation.  The  MAE  itself  does  not  support  full  body  weight;  rather,  the 
bones  of  the  user  actually  carry  the  load.  The  exoskeleton  merely  holds  the 
knee  so  it  does  not  collapse,  provides  support  at  the  thigh,  and  stabilizes  the 
foot.  A  mechanical  linkage  controls  the  angle  of  the  foot  according  to  the 
position  of  the  knee,  so  that  the  foot  forms  a  stable  base  of  support.  The 
same  linkage,  together  with  a  toe-lift  spring,  causes  the  correct  foot 
trajectory  for  ground  clearance  (dorsiflexion)  during  the  swing  phase.  For 
protection,  the  exoskeleton  mechanically  prevents  any  motion  outside  the 
natural  range,  while  an  allowance  in  the  linkage  allows  the  foot  to  extend 
beyond  the  ordinary  linkage-determined  walking  values.  When  seated,  the 
patient's  feet,  for  example,  rest  comfortably  and  naturally  flat  on  the  floor. 

Two  passive  prototypes  were  developed  during  this  project  and  the 
second  version  significantly  reduced  the  weight  and  bulk  of  the  MAE  to  allow 
use  when  worn  under  loose  clothing.  Passive  dynamic  MAEs  do  not  require 
batteries  or  other  power  sources,  however,  they  cannot  directly  augment 
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strength  to  allow  stair  climbing  or  rising  from  a  seated  position  without 
additional  injection  of  power.  These  activities  would  require  a  more  complex 
energy  storage  capability  or  powered  actuators. 

Future  improvements  to  the  passive  dynamic  MAE  systems  would 
include  linkages  to  generate  proper  pelvic  motion  and  support  and  a  stable, 
yet  compliant  knee  and  foot  suspension  for  more  dynamic  activity  power 
such  as  raising  or  lowering  the  knee  while  bearing  load  and  standing  from  a 
sitting  position  or  stair  climbing. 

Powered  Dynamic  MAE 

To  ensure  comfort  and  to  align  the  actuator  joints  to  those  of  the  user, 
the  initial  powered  dynamic  MAE  prototype  provides  numerous  position 
adjustments  for  the  exoskeleton  joints  and  the  body  connector  braces.  The 
ranges  of  adjustments  were  chosen  to  fit  the  10th  to  90th  percentile  of  users 
(Tilley,  2002;  NASA,  2006).  The  exoskeleton  includes  a  3-degree  of  freedom 
(DOF)  hip  joint  that  adjusts  to  a  wide  range  of  different  sized  individuals 
using  a  curved  roller  bearing,  which  locates  the  center  of  rotation 
approximately  at  the  user's  hip  joint.  Spring  mechanisms  allow  moderate 
misalignment  in  a  small  range  of  rotation  (±10  deg)  at  the  hip  joint  and 
provide  unidirectional  dorsiflexion  of  the  foot  at  the  ankle  to  lift  the  toe 
during  walking.  The  dynamic  lower  extremity  joints  were  modeled  as  fixed, 
single  axis  joints  such  that  each  actuator  axes  passes  through  the 
approximate  center  of  the  user's  corresponding  joint.  As  with  the  passive 
MAE,  the  powered  dynamic  MAE  does  not  support  the  weight  of  the  user;  the 
user's  weight  is  transmitted  through  his/her  bone  structure  to  the  ground. 
This  ensures  that  a  user's  bone  structure  experiences  normal  gravitational 
induced  loads  during  walking  to  reduce  bone  loss  caused  by  disuse.  The 
exoskeleton  does  support  its  own  weight  by  providing  a  load  path  through 
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the  leg  and  feet  structures  to  the  ground  and  the  user  provides  balance 
control  using  torso  movements  and  a  pair  of  arm  crutches. 

Rotary  Series  Elastic  Actuator  (RSEA) 

IHMC  chose  series  elastic  actuators  for  mobility  assistance  because  of 
their  force  control  capabilities  and  their  compliant  behavior  in  the  operational 
environment  (Sensinger  &  Weir,  2005).  The  key  element  of  a  series  elastic 
actuator  is  the  stiff  spring  placed  in  series  with  the  actuator.  The  actuator 
control  system  determines  the  actuator  output  force  from  the  spring 
constant  and  the  amount  of  compression  of  the  spring  (detected  by  a  high 
resolution  position  encoder).  Unlike  the  position  control  methods  used  with 
industrial  robots,  force  control  allows  the  system  to  react  quickly  to 
perturbations  or  obstacles  that  might  be  experienced  in  the  real  world. 
Rotary  Series  Elastic  Actuators  (RSEAs)  developed  specifically  for  this  project 
were  used  to  power  the  exoskeleton's  joints  as  compact,  efficient,  compliant 
joint  actuators  (Figure  10). 


Figure  10:  CAD  drawings  of  three  different  Rotary  Series  Elastic  Actuator  designs  tested  in 
this  project.  The  design  on  the  left  uses  a  single  cable  and  has  a  non-linear  torque  versus 
angular  deflection  profile.  The  design  in  the  middle  uses  two  cables  and  springs  and  has  a 
linear  torque  versus  angular  deflection  profile.  The  design  on  the  right  uses  cam  shaped 
surface  and  pair  springs  to  produce  a  non-linear  torque  versus  angular  deflection  profile. 

In  contrast  to  their  linear  counterparts,  rotary  actuators  do  not  require 
hard  anchor  points  against  which  to  exert  force  across  a  joint.  This  allows 
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the  use  of  soft,  compliant  interfaces  to  the  lower  extremities  and  torso  and 
tolerance  to  misalignments.  To  prevent  overextending  the  user's  joints,  the 
exoskeleton's  joints  have  been  fitted  with  mechanical  limit  stops  tested  to 
withstand  the  full  power  torque  capabilities  of  the  motors  and  amplifier 
setup.  In  addition  to  position  and  force  sensors  on  each  RSEA  actuator,  the 
MAE  has  foot  switches  at  the  heel  and  the  toes  to  detect  if  the  foot  is  on  the 
ground  and  enable  the  control  system  to  determine  whether  the  exoskeleton 
is  in  the  single  support  (only  one  foot  is  loaded),  double  support  (both  feet 
loaded)  or  toe  off  stage  of  the  gait  cycle. 

We  designed  three  different  Rotary  Series  Elastic  Actuators  and 
compared  their  performance.  The  design  that  exhibited  the  most  linear 
torque  versus  deflection  profile  also  demonstrated  the  best  torque  and 
position  closed-loop  bandwidth  performance,  and  following  mechanical 
design  optimization,  was  used  for  all  six  actuated  DOF  (see  Table  1). 


Table  I 

List  of  the  exoskeleton  degrees  of  freedom  and  their  control  method. 


Control 

Joint 

Method 

Range  of  Motion 

Hip  Pitch  (extension) 

Actuated 

+42°  (fwd),  -30°(back) 

Hip  Roll  (adduction) 

Actuated 

+25°  (out),  -30o  (in) 

Hip  Yaw  (rotation) 

Knee  Pitch  (extension) 

Passive 

Actuated 

±  10° 

+0  o,  -90  o 

Ankle  Pitch  (dorsiflexion  /  plantar 
flexion) 

Passive 

+20°  (up),  -35°  (down) 

Ankle  Roll  (inversion  /  eversion) 

Constrained 

N/A 

Ankle  Yaw  (rotation) 

Constrained 

N/A 

The  RSEA  control  system  can  use  both  low  and  high  impedance  within 
the  rated  bandwidth  and  torque  output  of  the  actuator,  enabling  strategies  of 
torque,  position,  or  impedance  control  or  combinations  thereof.  As  a 
performance  enhancement  device,  the  MAE  operates  in  torque  control  mode; 
for  disable  assist  applications,  the  MAE  uses  position  control  mode.  The 
exoskeleton  is  controlled  by  an  off-board  microprocessor  control  system,  and 
connects  via  a  multiwire  tether  for  power,  control  and  data  transfer. 
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Zero  Assistance  Control 

A  successful  MAE  device  should  assist  the  user  when  needed,  but  not 
obstruct  the  user  otherwise.  In  zero  assistance  control  mode,  the  IHMC  MAE 
performs  gravity  compensation  to  relieve  the  user  of  the  weight  of  the 
hardware,  but  not  to  provide  any  strength  augmentation.  While  walking  with 
a  regular  gait,  a  user's  joint  angles  were  recorded  using  a  ShapeWrapII 
(Measurand,  Inc.,  Fredericton,  NB,  Canada)  wireless,  wearable  motion 
tracking  system.  With  the  IHMC  MAE  in  this  mode  the  user  can  ambulate 
normally  while  feeling  little  or  no  resistance.  Figure  11  shows  a  comparison 
of  the  user's  hip  and  knee  flexion  joint  angles  while  walking  with  and  without 
the  IHMC  Mobility  Assist  Exoskeleton  in  zero  assistance  control  mode. 


*/.  Galt 


Figure  11:  Comparison  of  hip  and  knee  joint  data  without  exoskeleton  and  with 
exoskeleton  in  zero  assistance  mode  for  regular  walking.  Scale  and  offset  of  joint  angles 
show  minimal  changes  with  MAE  during  ambulation. 


Performance  Enhancement 

In  performance  enhancement  mode,  the  MAE  counters  the 
gravitational  forces  on  the  user,  as  well  as  any  loads  on  the  user,  including 
the  MAE  system  itself,  using  a  torque  control  mode  based  on  a  function  of 
the  gait  phase  of  the  leg  (swing  or  stance),  the  configuration  of  the  joints, 
and  the  desired  amount  of  assistance. 
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Gait  Rehabilitation 

The  IHMC  Mobility  Assist  Exoskeleton  (Figure  12)  has  great  potential 
for  gait  rehabilitation  because  of  its  high  fidelity  impedance  control  that  can 
be  incorporated  into  a  mobile  system.  This  allows  for  over  ground  gait 
training  during,  for  example,  rehabilitation  for  hemiparetic  stroke  survivors. 
The  MAE  control  software  can  define  the  behavior  of  each  joint  separately  to 
utilize  different  combinations  of  position  and  force  control. 


Figure  12:  IHMC  Mobility  Assist  Exoskeleton  modified  for  straight  line  walking.  The  hip  ab/ 
adduction  actuators  and  the  hip  internal/external  passive  rotation  degrees  of  freedom  have 
been  replaced  with  a  rigid  link. 

In  this  example,  the  MAE  assists  the  weak  leg  with  force  control  and 
provides  zero  assistance  to  the  strong  leg.  Dynamically  adjusting  the 
amount  of  assistance  provides  varying  levels  of  augmentation  (or  resistance) 
during  strength  rehabilitation  and  can  even  allow  a  user  with  lower  extremity 
paralysis  to  ambulate.  For  patients  with  no  motor  control  in  the  lower 
extremities,  the  exoskeleton  operates  in  disable  assist  mode,  generating 
position  trajectories  for  the  joints  while  the  user  provides  balance  through 
motions  of  the  torso  and  by  the  use  of  crutches. 

During  initial  testing  for  disable  assist  mode,  it  was  determined  that 
the  user's  center  of  mass  was  shifted  too  far  back.  In  order  move  the  center 
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of  mass  toward  the  users  center  of  gravity,  the  hip  ab/adduction  actuators 
and  the  passive  hip  internal/external  rotation  mechanism  were  replaced  with 
a  rigid  link.  Straight  line  walking  does  not  require  these  degrees  of  freedom 
so  this  change  does  not  affect  system  performance.  While  an  able-bodied 
user  ambulated  wearing  the  MAE  in  zero  assistance  control  mode,  joint 
positions  and  loads  on  the  feet  were  recorded  to  disk.  Various  types  of  gaits 
were  recorded,  including  static  and  dynamic  walks,  with  a  range  of  swing  leg 
ground  clearance  gaits.  For  evaluation  (with  able-bodied  users),  the 
actuators  function  under  position  control  and  move  the  joints  to  generate 
locomotion  according  to  trajectories  generated  from  the  recorded  data. 
Because  the  IHMC  Mobility  Assist  Exoskeleton  has  a  passive  ankle,  toe  off 
force  was  intentionally  limited  in  the  gait  recordings. 


Hip  Angla  and  Torque  for  Record  and  Playback 

~7>  0 


30 

20 


to 


0 


-30 


-to 


Time  (s) 


Figure  13:  Graph  of  hip  flexion  (left  graph)  and  knee  flexion  (right  graph)  angle  and  torque 
data  for  recording  and  playback  of  able-bodied  user  wearing  the  IHMC  Mobility  Assist 
Exoskeleton.  The  user  was  intentionally  walking  with  a  hyper  extended  support  knee  and 
exaggerated  knee  flexion  during  swing  for  large  ground  clearance.  During  recording,  both 
hip  and  knee  joint  torques  are  approximately  zero  because  the  actuators  were  in  zero 
assistance  mode.  During  playback,  the  user  tried  not  to  use  his  leg  muscles,  which  is  shown 
by  the  required  joint  torque  being  non-zero. 


Figure  13  shows  the  results  from  the  recording  and  playback  trials  for 


one  of  the  researchers  using  the  MAE  in  disable  assist.  During  playback, 
negligible  error  manifested  between  the  actual  and  desired  joint  angles. 
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Disable  assist  mode  provides  dynamic  collaboration  between  the  user  and 
the  exoskeleton  through  adjustable  playback  speed  and  playback  pause  at 
any  point  during  the  walking  gait.  The  user  must  anticipate  the  next  move  of 
the  exoskeleton's  lower  extremities  and  adjust  his/her  torso  position  to 
unload  the  upcoming  swing  leg  during  double  support  phase.  Four  IHMC 
researchers  tested  the  disable  assist  mode  using  the  same  recorded  disable 
assist  appropriate  gait  data  file  that  featured  small  steps,  very  little  knee 
flexion  during  support,  and  large  knee  flexion  during  swing.  Within  thirty 
minutes  of  practice,  all  of  the  users  could  walk  a  short  distance  at  full 
playback  speed  and  provide  balance  support  with  forearm  crutches. 

Demonstrations 

All  demonstrations  were  conducted  primarily  for  the  edification  of  the 
users  to  give  them  a  "sneak  peak"  at  future  technologies  that  may  soon  be 
available  to  them.  Their  subjective,  critical  comments  helped  guide  later 
development.  Each  servicemember  was  fully  informed  about  the 
technologies  and  risks  involved  in  the  demonstrations.  Care  was  taken 
during  briefings  to  ensure  that  their  expectations  of  the  technologies 
matched  current  device  capabilities.  All  provided  permission  to  use  their 
likenesses  in  reports  and  presentations  and  no  data  was  collected  or 
recorded  during  the  demonstrations.  Early  brain  plasticity,  previous  sensory 
capabilities,  technical  savviness  and  eagerness  to  regain  social  interactions 
likely  contributed  to  rapid  assimilation  and  mastery  of  the  sensory 
technologies  shown  by  these  participants.  Prior  research  indicates  that 
congenitally  blind  civilians  or  those  for  whom  decades  have  past  since  injury 
require  much  more  time  and  cognitive  effort  to  reach  similar  outcomes. 

Sensory  Substitution  Technologies 

Four  profoundly  blind  servicemember  or  veteran  (3  Army,  1  Marine) 
participants  evaluated  the  sensory  substitution  systems.  Each  progressed 
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rapidly  through  the  training  and  demonstrations  for  vision  sensory 
substitution,  exhibiting  their  high  level  of  motivation  and  the  intuitiveness  of 
the  research  technologies.  For  the  BrainPort®  vision  device,  we  based  our 
initial  training  protocol  on  one  which  is  currently  being  employed  at  Wicab. 
In  our  test  room,  a  high  contrast  environment  was  created  using  black 
backgrounds  overlaid  by  white  objects  or  vice  versa  (Figure  14). 


Figure  14:  Blind  servicemembers  training  on  BrainPort®  Vision  system,  left,  Reading  a 
word,  and,  right,  Identifying  and  picking  up  a  ball. 


To  orient  subjects  with  the  device,  we  first  introduced  a  white  pipe 
(approximately  1  meter)  in  static  vertical,  horizontal  and  diagonal  positions. 
User's  were  asked  to  identify  the  positions  and  use  the  BrainPort®  zoom 
function  to  perceive  depth.  This  was  followed  by  the  presentation  of  the 
letter  C,  then  shapes  such  as  circles,  squares,  triangles  and  diamonds.  For 
these  first  tasks,  the  participants  were  encouraged  to  touch  and  explore  the 
pipe,  shapes  and  letters  to  help  them  correlate  the  novel  tongue  sensations 
with  more  familiar  stereognosis.  Reading  proficiency  was  then  tasked,  as 
subjects  were  asked  to  learn  the  distinguishing  characteristics  of  each  letter 
and  then  read  words  printed  on  a  sheet;  most  were  able  to  read  3-5  word 
sentences  at  a  reasonable  rate  (Figure  14).  To  test  more  advanced  depth 
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perception,  three  balls  of  varying  sizes  were  placed  in  front  of  the  subject  in 
staggered  position,  and  they  were  asked  to  identify  and  pick  up  the  largest 
(or  smallest)  of  the  group.  The  static  orientation  training  concluded  when 
each  servicemember  was  able  to  correctly  identify  all  objects  or  words,  and 
do  so  at  a  comfortable  pace.  Although  the  time  to  learn  this  complement  of 
tasks  varied  among  the  group,  no  single  servicemember  took  longer  than 
three  hours. 

For  dynamic  tasking,  each  servicemember  learned  to  catch  a  ball  rolled 
from  across  a  table  (Figure  14).  Each  participant  then  exited  the  training 
room  and  walked  through  hallways  and  doorways.  Using  either  a  white 
centerline  stripe  on  the  floor  or  a  safety  orange  runner  with  a  black 
centerline  stripe  providing  a  high  contrast  guidance  cue,  each 
servicemember  was  able  to  walk  comfortably  without  any  other  means  of 
assistance.  We  then  moved  outside  the  building,  where  black  painted  stair 
edges  allowed  each  individual  to  walk  up  and  down  steps  unassisted.  Upon 
reaching  the  lower  level  parking  lot,  the  servicemembers  were  able  to  follow 
parking  lines  and  avoid  static  objects,  such  as  automobiles  (Figure  15).  To 
enhance  perception  of  foot  position,  we  fitted  battery  powered  white  LEDs  to 
the  top  of  each  participant's  shoes  (some  shoe  colors  lacked  contrast  with 
the  floor,  pavement  or  stairs). 

Other  tasks,  such  as  reading  an  eye  chart,  playing  tic-tac-toe,  or  trying 
to  identify  foods  or  facial  characteristics  were  added  as  we  progressed  and 
individuals  sought  more  challenging  assignments.  One  servicemember  (a 
former  U.  S.  Army  sniper)  managed  to  read  a  standard  Snellen  eye  chart  at 
an  acuity  level  of  20/80,  while  another  identified  that  his  infant  son's  hair 
was  tousled  from  the  visually  substituted  sensation  on  his  tongue. 
Interestingly,  since  the  child  had  been  born  post-deployment,  it  was  this 
servicemember's  first  "visual"  perception  of  his  son  (Figure  15). 
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Figure  15:  Dynamic  and  ad-libbed  BP-WAVE  II  activities  performed  during  technology 
demonstrations  with  recently  blinded  servicemembers.  left  to  right,  top  to  bottom,  walking 
down  stairs,  navigating  office  spaces,  locating  open  parking  stalls,  noticing  infant's  hair,  and 
reading  eye  charts  to  20/80,  evaluating  TTI  for  peripheral  tactile  cues. 


Today's  servicemembers  rely  on  the  Internet  for  many  social  and 


educational  activities  and  loss  of  vision  severely  restricts  this  experience  as 


the  Internet  is  designed  for  those  with  sight,  favoring  images  over  textual 
information.  To  address  these  concerns,  IHMC  integrated  the  BrainPort® 


with  eye/head  tracking  equipment  to  demonstrate  that  a  direct  connection 
between  the  tactile  interface  and  a  computer  graphical  display  can  provide  a 


28 


superior  qualitative  experience  by  bypassing  the  limitations  of  cameras  (e.g., 
glare,  changing  lighting  conditions,  focus  and  unintentional  movements  of 
the  head).  We  prototyped  three  methods  of  tactile  computer  interactions 
that  mimic  sighted  interactions  (Figure  16). 


Figure  16:  Direct  computer  tactile  (BrainPort®)  interface  demonstrations,  left,  participant 
prepares  to  use  eye/head-tracking  apparatus  to  determine  point  of  gaze  despite  the  users' 
prosthetic  eyes.  The  software  determines  if  the  participant  is  looking  at  the  screen  and 
returns  a  100-200  pixel  sample  of  the  screen  image;  right,  When  the  user  touches  the 
screen,  the  system  automatically  switches  to  providing  the  subsection  of  the  screen  image 
underneath  the  fingertip  to  the  tongue. 

All  of  these  participants  mastered  the  entire  suite  of  sensory 
substitution  tasks  in  less  than  8  hours  total  training  time,  significantly  faster 
than  civilian  congenitally  blind  individuals  or  those  who  have  been  blind  for 
many  decades.  This  indicates  that  previous  sightedness  provides  a  reference 
when  interacting  with  common  objects  and  that  brain  plasticity  likely  fades 
as  the  individual  learns  alternative  mechanisms  of  coping  with  their 
blindness. 

Mobility  Assistance  Exoskeleton 

We  demonstrated  the  passive  dynamic  MAE  to  one  servicemember 
(Air  Force)  who  had  TBI  effects  following  an  intracranial  bleed  from  a 
ruptured  arteriovenous  malformation.  Following  a  three-month  coma,  this 
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individual  awoke  unable  to  speak  and  with  a  left  sided  hemiplegia.  Two 
months  following  the  coma  he  reacquired  speech  and  by  the  time  he  arrived 
for  his  demonstration,  had  learned  to  walk  again.  However,  he  required  a 
cane  and  ambulated  slowly  and  deliberatively  to  ensure  that  he  positioned 
his  weak,  paretic  leg  appropriately  to  support  his  weight  on  each  step. 
Following  a  ten-minute  training  session  with  the  passive  dynamic  MAE 
prototype  on  an  indoor  ambulation  course,  this  individual  gained  sufficient 
confidence  in  his  gait  that  he  was  able  to  walk  at  a  normal  pace  with  cane 
after  doffing  the  MAE  (Figure  17). 


Figure  17:  left,  Hemiplegic  servicemember  using  passive  dynamic  MAE;  right,  Walking 
unassisted  without  MAE,  cane  or  assistance  at  near  normal  pace 

After  completing  the  ambulation  course  with  his  cane,  he  set  it  down 
and  walked  the  course  without  it  at  a  near  normal  pace.  This  was  the  first 
time  he  had  walked  unassisted  since  his  injury.  We  believe  that  his  slow  gait 
was  due  to  uncertainty  in  the  stability  of  his  foot  plant  on  each  step  as  a 
result  of  the  diminished  sensation  and  strength  available  in  his  left  leg.  By 
ensuring  that  his  weight  would  be  supported  on  each  step,  the  MAE  may 
have  allowed  him  to  place  greater  trust  in  his  existing  cerebellar  gait 
patterns  and  believe  that  his  normal  hip  swing  would  place  his  foot  in  the 
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proper  position  for  the  next  step,  even  if  he  could  not  perceive  its  position 
accurately. 

Balance  Rehabilitation 

We  are  supporting  ongoing  studies  using  Wicab,  Inc.,  BrainPort® 
Balance  Devices  by  leasing  or  loaning  BrainPort®  units  to  the  Atlanta 
Veterans  Affairs  Hospital  and  the  Naval  Medical  Center-San  Diego  for  use  in 
their  research  protocols  for  balance  retraining  during  recovery  from 
traumatic  brain  injury  (TBI).  Combined,  they  have  recruited  20  participants 
into  their  protocols  which  are  ongoing  and  separate  from  this  project. 
Conversations  with  the  principal  investigators  of  those  protocols  have 
indicated  that  the  BrainPort®  balance  substitution  training  appears  well 
tolerated  and  effective. 

Public  Engagement 

Demonstrations  were  provided  for  able-bodied  interested  military  and 
civilian  individuals  at  IHMC.  These  included  a  project  overview,  goals  and 
technology  demonstrations  for  Commander  U.S.  Army  Medical  Command/ 
Acting  The  Surgeon  General,  MG  Gale  Pollock,  and  Commander,  Naval 
Hospital  Pensacola,  CAPT  Kevin  Berry.  A  local  television  channel  news  report 
of  the  event,  which  was  also  supported  by  Wicab,  Inc.,  is  available  at: 
http://www.ihmc.us:  16080/araj/Movies/wearTV3BrainPort20070816.mov.  In 

addition,  a  number  of  clinicians  (neurologists,  otolaryngologists,  orthotists, 
etc.)  participated  in  shortened  demonstrations  to  help  them  understand  the 
novel  technologies.  Technology  demonstrations  were  also  provided  for  two 
local  civilians  with  partial  vision  loss  due  to  stroke  and  one  profoundly  blind 
individual  at  their  request  after  they  learned  of  the  technologies.  One 
paraplegic  individual  provided  a  subjective  evaluation  of  the  powered 
dynamic  MAE,  but  lacked  sufficient  upper  body  strength  to  participate  in  an 
actual  demonstration. 
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e.  Major  Problems/Issues 

We  were  not  able  to  develop  our  initial  contacts  at  Walter  Reed  Army 
Medical  Center  and  the  National  Naval  Medical  Center  due  to  changes  in 
personnel;  however,  we  developed  alternate  military  contacts  to  provide 
demonstrations  to  wounded  servicemembers  during  this  project. 

Initial  attempts  to  provide  wide  field  of  view  for  peripheral  vision 
substitution  centered  on  identification  of  appropriate  high  resolution  cameras 
and  wide  angle  or  panoramic  lenses.  These  efforts  were  stymied  by  the 
weight  and  bulk  of  the  optics  as  well  as  the  amount  of  computation  power 
required  to  process  full  screen  video.  The  current  IR  emitter/detector 
approach  is  much  more  efficient  computationally,  and  will  work  in  the  dark 
(the  blind  often  do  not  turn  on  the  lights  in  a  dark  room),  but  the  late  switch 
to  this  concept  prevented  us  from  completing  the  prototype  before  the  end 
of  the  project. 

f.  Technology  Transfer 

Wicab,  Inc.,  has  developed  a  production  model  BrainPort®  for  vision 
substitution  that  utilizes  the  25x25  arrays  and  is  now  available  for 
investigational  use. 

ForeThought  Development,  LLC,  is  currently  developing  a  new  version 
of  the  VideoTact  with  significantly  reduced  size  and  weight.  This  will  use  only 
currently  available  components  and  transition  to  surface  mount  components. 
The  new  VideoTact  will  fit  under  the  user's  clothing  and  operate  completely 
wirelessly. 

g.  Foreign  Collaborations  and  Supported  Foreign  Nationals 

Three  foreign  nationals,  Cecilia  Aguero  (France),  David  Lecoutre 
(France)  and  Elsa  Fouragnan  (France),  supported  this  project  as  graduate 
student  assistants.  All  were  Masters  candidates  in  Cognitive  Engineering 
from  the  Institut  de  Cognitique,  Universite  de  Bordeaux-II,  participating  as 
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research  assistant  interns  as  part  of  their  curriculum  and  worked  on 
developing  sensory  substitution  systems  including  software  coding  and 
development  of  training  paradigms. 

Jerryll  Noorden  (The  Netherlands),  Hian  Kai  Kwa  (Singapore),  Victor 
Ragusila  (Canada)  were  mechanical  engineering  research  associates  who 
helped  design,  fabricate,  test  and  evaluate  the  rotary  series  elastic  actuators 
and  the  mobility  assist  exoskeletons. 

h.  Productivity 

a.  Refereed  journal  articles:  None 

b.  Non-refereed  significant  publications:  None 

c.  Books  or  chapters:  None 

d.  Tech  reports:  None 

e.  Workshops/  conferences: 

Raj  A.  K.  &  Cameron,  J.  (2008)  Sensory  Substitution  Technologies  for 
TBI.  Naval  Hospital  Pensacola  Traumatic  Brain  Injury  (TBI) 
Seminar.  Pensacola,  FL,  13  March. 

Kwa,  H.  K.,  Noorden,  J.  H.,  Missel,  M.,  Craig,  T.,  Pratt,  J.  E.,  &  Neuhaus, 
P.  D.  (2009)  Development  of  the  IHMC  Mobility  Assist 
Exoskeleton.  IEEE  International  Conference  on  Robotics  and 
Automation.  2556-2562. 

f.  Patents:  None 

g.  Awards  honors:  None 
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Acronyms: 

ADL: 

Activities  of  Daily  Living 

AMI: 

Adaptive  Multi-Agent  Integration 

CAD: 

Computer-Aided  Design 

CG: 

Center  of  Gravity 

DOF: 

Degree  of  Freedom 

FPGA: 

Field  Programmable  Gate  Array 

GUI: 

Graphical  User  Interface 

IHMC: 

Florida  Institute  for  Human  and  Machine  Cognition 

IOD: 

Intra-Oral  Device 

IR: 

Infrared 

JNI: 

Java  Native  Interface 

LED: 

Light  Emitting  Diode 

LLC: 

Limited  Liability  Company 

MAE: 

Mobility  Assist  Exoskeleton 

MG: 

Major  General 

ONR: 

Office  of  Naval  Research 

PC: 

Personal  Computer 

PCI: 

Peripheral  Component  Interconnect 

PIM: 

Process  Integrated  Mechanism 

RSEA: 

Rotary  Series  Elastic  Actuator 

SSI: 

Sensory  Substitution  Interface 

TBI: 

Traumatic  Brain  Injury 

TCP/IP: 

Transmission  Control  Protocol/Internet  Protocol 

TDU: 

Tongue  Display  Unit 

TSAS: 

Tactile  Situation  Awareness  System 

TTI: 

Tactile  Torso  Interface 
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